Skip to content

SceneDataTransformer: Check metadata when choosing to emit transformation data#1408

Open
idastambuk wants to merge 1 commit intomainfrom
idastambuk/querytransformer-fix
Open

SceneDataTransformer: Check metadata when choosing to emit transformation data#1408
idastambuk wants to merge 1 commit intomainfrom
idastambuk/querytransformer-fix

Conversation

@idastambuk
Copy link
Copy Markdown
Contributor

Summary

Fixes this escalation: https://github.com/grafana/support-escalations/issues/21201

The problem was that if a panel referenced other dashboard panels that contain transformation, turning off and on those transformations made the query hang and the data never to come in.

Screenshot 2026-03-26 at 14 00 34

With Graft, I managed to bisect and pinpont the commit that broke this: #1370

The PR changed SceneQueryRunner to preserve series and annotations references when frames were unchanged. As a result, downstream consumers such as SceneDataTransformer began receiving normal request/state transitions with reused frame references.
That exposed an incorrect assumption in SceneDataTransformer: it treated unchanged frame references as meaning the entire transformed result could be reused without further propagation. In practice, the incoming PanelData could still carry meaningful top-level changes such as:

  • a new requestId
  • a new state, such as Done -> Loading -> Done
  • updated errors
  • updated timeRange

When that happened, SceneDataTransformer's cache-hit path would skip transformation work, but it would also skip emitting a refreshed transformed result to _results. Dashboard-datasource consumers subscribed to the transformed provider therefore never observed the new request lifecycle and could remain stuck loading.

Fix

SceneDataTransformer.haveAlreadyTransformedData() now handles same-reference cache hits by preserving the already transformed series and annotations while checking whether meaningful top-level metadata changed.
When the source reuses frame references, the transformer now:

  • builds a merged PanelData using the incoming source metadata and the cached transformed frames
  • compares the current transformed state to the merged result
  • only updates state and emits to _results when meaningful metadata actually changed
    The metadata check currently covers:
  • state
  • request?.requestId
  • error
  • errors
  • timeRange

In the buggy workflow from the customer instance, the transformed data path received a new upstream lifecycle update without getting new frame references. The series / annotations references stayed the same, but the top-level PanelData metadata changed, most importantly state and request?.requestId. That meant the update was previously treated as an already-transformed cache hit and silently swallowed. The new metadata condition specifically catches that case and propagates the updated lifecycle information without forcing a full re-transformation.
Why This Works
This keeps the performance benefit introduced by the PR: unchanged frames still do not get retransformed unnecessarily.
At the same time, it ensures that transformed data providers continue to propagate important request lifecycle updates to downstream consumers when those updates matter. This prevents dashboard-datasource panels from getting stuck on stale transformed state while avoiding the broader side effect of emitting on every same-reference wrapper update.

In short, the fix preserves the identity-reuse optimization while making SceneDataTransformer aware that same frame references do not necessarily mean unchanged top-level PanelData metadata.

testing

I could only reproduce this in the customer instance, not locally. But with Graft I managed to verify the fix

@idastambuk idastambuk requested review from a team, harisrozajac and ifrost and removed request for a team March 30, 2026 08:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant